{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 混淆矩阵\n",
    "\n",
    "|真实预测|0|1|\n",
    "|:--:|:--:|:--:|\n",
    "|0|TN|FP|\n",
    "|1|FN|TP|\n",
    "\n",
    "T/F代表预测结果是true or false，N/P代表预测为阴性或是阳性。\n",
    "\n",
    "因此 TN 表示真实类别是 Negtive，预测结果也是 Negtive 的数量； FP 表示真实类别是 Negtive，预测结果是 Positive 的数量； FN 表示真实类别是 Positive，预测结果是 Negtive 的数量；TP 表示真实类别是 Positive，预测结果也是 Positive 的数量。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "def confusion_matrix(y_true, y_predict):\n",
    "    '''\n",
    "    构建二分类的混淆矩阵，并将其返回\n",
    "    :param y_true: 真实类别，类型为ndarray\n",
    "    :param y_predict: 预测类别，类型为ndarray\n",
    "    :return: shape为(2, 2)的ndarray\n",
    "    '''\n",
    "\n",
    "    #********* Begin *********#\n",
    "        \n",
    "    return np.array([[np.sum((y_true == 0) & (y_predict == 0)), np.sum((y_true == 0) & (y_predict == 1))],[np.sum((y_true == 1) & (y_predict == 0)), np.sum((y_true == 1) & (y_predict == 1))]])\n",
    "\n",
    "    #********* End *********#\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_true = np.array([1,1,1,0,0])\n",
    "y_pred = np.array([0,1,1,1,0])\n",
    "confusion_matrix(y_true, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 精确率和召回率\n",
    "\n",
    "**精准率(Precision)** 指的是模型预测为 Positive 时的预测准确度，其计算公式如下：\n",
    "\n",
    "$$\n",
    "Precision = \\frac{TP}{TP+FP}\n",
    "$$\n",
    "\n",
    "**召回率(Recall)** 指的是我们关注的事件发生了，并且模型预测正确了的比值，其计算公式如下：\n",
    "\n",
    "$$\n",
    "Recall = \\frac{TP}{FN+TP}\n",
    "$$\n",
    "\n",
    "F1 Score 是统计学中用来衡量二分类模型精确度的一种指标。它同时兼顾了分类模型的准确率和召回率。 F1 Score 可以看作是模型准确率和召回率的一种加权平均，它的最大值是 1，最小值是 0。其公式如下：\n",
    "\n",
    "$$\n",
    "F1 = \\frac{2*precision*recall}{precision+recall}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "def precision_score(y_true, y_predict):\n",
    "    '''\n",
    "    计算精准率并返回\n",
    "    :param y_true: 真实类别，类型为ndarray\n",
    "    :param y_predict: 预测类别，类型为ndarray\n",
    "    :return: 精准率，类型为float\n",
    "    '''\n",
    "\n",
    "    #********* Begin *********#\n",
    "    TP = np.sum((y_true == 1) & (y_predict == 1))\n",
    "    FP = np.sum((y_true == 0) & (y_predict == 1))\n",
    "    return TP / (TP + FP)\n",
    "    #********* End *********#\n",
    "\n",
    "\n",
    "def recall_score(y_true, y_predict):\n",
    "    '''\n",
    "    计算召回率并召回\n",
    "    :param y_true: 真实类别，类型为ndarray\n",
    "    :param y_predict: 预测类别，类型为ndarray\n",
    "    :return: 召回率，类型为float\n",
    "    '''\n",
    "\n",
    "    #********* Begin *********#\n",
    "    TP = np.sum((y_true == 1) & (y_predict == 1))\n",
    "    FN = np.sum((y_true == 1) & (y_predict == 0))\n",
    "    return TP / (TP+FN)\n",
    "    #********* End *********#\n",
    "\n",
    "def f1_score(precision, recall):\n",
    "    '''\n",
    "    计算f1 score并返回\n",
    "    :param precision: 模型的精准率，类型为float\n",
    "    :param recall: 模型的召回率，类型为float\n",
    "    :return: 模型的f1 score，类型为float\n",
    "    '''\n",
    "\n",
    "    #********* Begin *********#\n",
    "    return (2*precision*recall) / (precision + recall)\n",
    "\n",
    "    #********* End ***********#"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ROC与AUC曲线\n",
    "\n",
    "ROC曲线描述的 TPR （ True Positive Rate ）与 FPR （ False Positive Rate ）之间关系的曲线。\n",
    "\n",
    "TPR 与 FPR 的计算公式如下：\n",
    "\n",
    "$$\n",
    "TPR = \\frac{TP}{TP+FN}\n",
    "$$\n",
    "\n",
    "$$\n",
    "FPR = \\frac{FP}{FP+TN}\n",
    "$$\n",
    "\n",
    "其中 TPR 的计算公式您可能有点眼熟，没错！就是召回率的计算公式。也就是说 TPR 就是召回率。所以 TPR 描述的是模型预测 Positive 并且预测正确的数量占真实类别为 Positive 样本的比例。而 FPR 描述的模型预测 Positive 并且预测错了的数量占真实类别为 Negtive 样本的比例。\n",
    "\n",
    "很明显模型的 AUC 越高，模型的二分类性能就越强。AUC 的计算公式如下：\n",
    "\n",
    "$$\n",
    "AUC = \\frac{\\sum_{i \\in positiveclass} rank_i - \\frac{M*(M+1)}{2}}{M+N}\n",
    "$$\n",
    "\n",
    "其中 M 为真实类别为 Positive 的样本数量，N 为真实类别为 Negtive 的样本数量。ranki 代表了真实类别为 Positive 的样本点额预测概率从小到大排序后，该预测概率排在第几。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "def calAUC(prob, labels):\n",
    "    '''\n",
    "    计算AUC并返回\n",
    "    :param prob: 模型预测样本为Positive的概率列表，类型为ndarray\n",
    "    :param labels: 样本的真实类别列表，其中1表示Positive，0表示Negtive，类型为ndarray\n",
    "    :return: AUC，类型为float\n",
    "    '''\n",
    "\n",
    "    #********* Begin *********#\n",
    "    tab = np.array([prob, labels])\n",
    "    tab = np.array([prob, labels]).T\n",
    "    assert tab.shape == (len(prob), 2)\n",
    "\n",
    "    M = np.sum((labels == 1))\n",
    "    N = len(labels) - M\n",
    "\n",
    "    # 按照第一列从小到大排序\n",
    "    tab = tab[np.argsort(tab[:,0])]\n",
    "    mask = np.ones_like(labels)\n",
    "    mask = (mask == tab[:, -1])\n",
    "    # np.nonzeros取出的是一个元组\n",
    "    rank = np.sum(np.nonzero(mask)[0] + 1)\n",
    "\n",
    "    return (rank  - M*(M+1)/2) / (M*N)\n",
    "    #********* End *********#\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prob = np.array([0.1, 0.4, 0.3, 0.8])\n",
    "labels = np.array([0, 0, 1, 1])\n",
    "calAUC(prob=prob, labels=labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 使用sklearn中性能指标的分类\n",
    "- accuracy_score(y_true, y_pred)\n",
    "- precision_score(y_true, y_pred, pos_label) #pos_label：用什么值表示 Positive，默认为 1。\n",
    "- recall_score(y_true, y_pred, pos_label) #pos_label：用什么值表示 Positive，默认为 1。\n",
    "- f1_score(y_true, y_pred, pos_label) #pos_label：用什么值表示 Positive，默认为 1。\n",
    "- roc_auc_score(y_true, y_score) # y_score：为模型预测样本为 Positive 的概率，类型为一维的 ndarray 或者 list。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, roc_auc_score\n",
    "\n",
    "def classification_performance(y_true, y_pred, y_prob):\n",
    "    '''\n",
    "    返回准确度、精准率、召回率、f1 Score和AUC\n",
    "    :param y_true:样本的真实类别，类型为`ndarray`\n",
    "    :param y_pred:模型预测出的类别，类型为`ndarray`\n",
    "    :param y_prob:模型预测样本为`Positive`的概率，类型为`ndarray`\n",
    "    :return:\n",
    "    '''\n",
    "\n",
    "    #********* Begin *********#\n",
    "    accuracy = accuracy_score(y_true, y_pred)\n",
    "    precision = precision_score(y_true, y_pred)\n",
    "    recall = recall_score(y_true, y_pred)\n",
    "    f1 = f1_score(y_true, y_pred)\n",
    "    roc_auc = roc_auc_score(y_true, y_prob)\n",
    "    return accuracy, precision, recall, f1, roc_auc\n",
    "    #********* End *********#\n",
    "    "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "newd2l",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.19"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
